Skip to main content

CPU Challenges

Study Roadmap for CPU in Linux​

Exercise 1: Monitor Basic CPU Usage​

  • Objective: Get a baseline of your system's CPU performance.
  • Tools: top, htop
  • Instructions:
    1. Run top or htop to monitor real-time CPU usage.
    2. Observe the CPU usage per core and the load average.
    3. Record metrics over a period of time and identify any trends.
Notes:​
  • Running top gives us some interesting information
Tasks: 132 total,   1 running, 131 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.5 us,  3.0 sy,  0.0 ni, 95.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   3910.6 total,   1062.8 free,    334.9 used,   2512.9 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   3383.7 avail Mem
  1. The total amount of processes in the machine - and even separated by status
  2. The CPU % usage spliited by `user, system, ni(user defined priorities),idle %, waiting-time (for IO), hi (time for hardware handling CPU time), si (code calling interruption), st (stolen machine - involuntary wait due to hypervisors)
  3. Memory and Swap Memory. Reference: https://unix.stackexchange.com/questions/18918/linux-top-command-what-are-us-sy-ni-id-wa-hi-si-and-st-for-cpu-usage
  • Running htop gives us almost the same inputs but with better visibility top-print

Exercise 2: Analyze Process CPU Usage​

  • Objective: Understand which processes are consuming CPU resources.
  • Tools: ps, top
  • Instructions:
    1. Use ps aux --sort=-%cpu to list processes sorted by CPU usage.
    2. Identify the top CPU-consuming processes.
    3. Use top to see dynamic changes in CPU usage and how processes behave over time.

Notes:​

Exercise 3: Measure CPU Performance with mpstat​

  • Objective: Get detailed CPU statistics over time.
  • Tools: mpstat
  • Instructions:
    1. Run mpstat -P ALL 1 to monitor CPU usage by each core.
    2. Observe idle time, user time, and system time across different cores.
    3. Analyze how CPU usage changes during different workloads.

Note​

  • Running mpstat -P ALL 1 lists CPU usage per core each 1s

Exercise 4: Perform CPU Stress Testing​

  • Objective: Simulate high CPU load and observe system behavior.
  • Tools: stress, stress-ng
  • Instructions:
    1. Use stress --cpu 4 --timeout 30 to create CPU load on 4 cores for 30 seconds.
    2. Monitor the impact on system responsiveness and CPU metrics using htop or top.
    3. Experiment with different numbers of CPU cores and durations.

Exercise 5: Analyze CPU Scheduling with perf​

  • Objective: Gain insights into scheduling events and CPU performance.
  • Tools: perf
  • Instructions:
    1. Run perf top to see a real-time view of CPU usage by functions.
    2. Run perf record -a followed by a workload (e.g., a stress test).
    3. Use perf report to analyze which functions were consuming the most CPU cycles.

Notes:​

Running perf top gives us some cool insights into the usability. I've ran it while stress-ng --cpu 1 --timeout 30 was active. What we could learn from the input is:

  64.40%  swapper          [kernel.kallsyms]         [k] default_idle_call
   6.24%  stress-ng        libm.so.6                 [.] 0x0000000000053e78
   5.90%  stress-ng        stress-ng                 [.] 0x0000000000122288
   1.67%  stress-ng        stress-ng                 [.] 0x000000000012228c
   1.62%  stress-ng        libm.so.6                 [.] __sqrtl_finite
   1.42%  swapper          [kernel.kallsyms]         [k] __softirqentry_text_start
   1.34%  stress-ng        libm.so.6                 [.] 0x0000000000053e7c
   0.77%  sshfs            [kernel.kallsyms]         [k] __wake_up_common_lock
   0.59%  perf             [kernel.kallsyms]         [k] __wake_up_common_lock
   0.52%  swapper          [kernel.kallsyms]         [k] do_idle
   0.47%  swapper          [kernel.kallsyms]         [k] cpuidle_idle_call

The swapper process is responsible for making the CPU idle. In this case, 64% of the time CPU was idle. To understand how much CPU % a process is using, we need to sum all the entries of it. For stress-ng it is around ~15%. We can notice that libm.so.6 is the main culprit of load - with an function call named __sqrtl_finite. We are assuming this since we don't have enough info about the code.

Exercise 6: Investigate Context Switching​

  • Objective: Understand how context switching affects CPU performance.
  • Tools: pidstat, vmstat
  • Instructions:
    1. Run pidstat -w 1 to monitor context switches for processes.
    2. Use vmstat 1 to observe context switches at the system level.
    3. Identify processes with high context switch rates and analyze their behavior.

Exercise 7: Examine CPU Core Affinity​

  • Objective: Learn how CPU affinity affects performance.
  • Tools: taskset, htop
  • Instructions:
    1. Use taskset to run a CPU-bound process on a specific core (e.g., taskset -c 0 ./your_program).
    2. Monitor CPU usage in htop to see how it behaves when pinned to a single core.
    3. Experiment with different core affinities and observe performance changes.

Exercise 8: Investigate CPU Thermal Throttling​

  • Objective: Understand how thermal throttling affects CPU performance.
  • Tools: lm-sensors, cpufrequtils
  • Instructions:
    1. Install lm-sensors and configure it to monitor CPU temperatures.
    2. Run a stress test while monitoring temperature and CPU frequency.
    3. Observe how high temperatures lead to throttling and impact performance.

Exercise 9: Analyze CPU Load Balancing​

  • Objective: Study how the Linux scheduler balances CPU loads.
  • Tools: sar, htop, dstat
  • Instructions:
    1. Use sar -P ALL 1 to monitor CPU usage across cores.
    2. Run workloads that generate different levels of CPU usage.
    3. Observe how the scheduler redistributes workloads among CPU cores.

Exercise 10: Long-term CPU Monitoring and Reporting​

  • Objective: Set up a system to collect and analyze CPU statistics over time.
  • Tools: sar, sysstat
  • Instructions:
    1. Install sysstat and enable data collection.
    2. Use sar -u 1 to collect CPU usage statistics over time.
    3. Analyze the collected data to identify trends, peak usage times, and potential bottlenecks.

Final Thoughts​

Make sure to document your findings, observations, and any surprising results for each exercise. This structured approach will help you develop a comprehensive understanding of CPU performance and scheduling in Linux, allowing you to effectively analyze and optimize CPU-related issues in your system. Enjoy your studies!

Advanced Section

Challenge 1: Analyze Multi-threaded Performance​

  • Objective: Measure the performance impact of multi-threading on CPU utilization.
  • Instructions:
    1. Create a multi-threaded application that performs heavy computations.
    2. Use perf to analyze CPU cycles, cache hits/misses, and context switches.
    3. Compare the performance with a single-threaded version.

Challenge 2: Investigate CPU Cache Effects​

  • Objective: Understand how cache levels (L1, L2, L3) affect performance.
  • Instructions:
    1. Write benchmarks to access data in different cache levels (e.g., large arrays).
    2. Use perf to measure cache hits and misses.
    3. Analyze how data locality impacts performance.

Challenge 3: Measure Context Switch Overhead​

  • Objective: Analyze the overhead of context switching on system performance.
  • Instructions:
    1. Create a scenario with high context switching (many short-lived processes).
    2. Use pidstat to measure context switches per second.
    3. Compare CPU utilization with and without context switch-heavy workloads.

Challenge 4: Explore CPU Scheduling Algorithms​

  • Objective: Compare different CPU scheduling algorithms (CFS, real-time).
  • Instructions:
    1. Create workloads that require real-time processing.
    2. Switch between scheduling policies using chrt and nice.
    3. Measure performance impacts on latency and throughput.

Challenge 5: Simulate CPU Contention​

  • Objective: Observe the effects of CPU contention on performance.
  • Instructions:
    1. Run multiple CPU-intensive processes on the same CPU core.
    2. Monitor CPU usage and process states with top or htop.
    3. Analyze the impact of contention on overall system performance.

Challenge 6: Investigate Power Management​

  • Objective: Understand how CPU power management affects performance.
  • Instructions:
    1. Use cpupower to change CPU frequency scaling governors (performance, powersave).
    2. Measure performance under different governors using a CPU benchmark.
    3. Analyze trade-offs between power savings and performance.

Challenge 7: Explore Hyperthreading Effects​

  • Objective: Measure the impact of hyperthreading on CPU performance.
  • Instructions:
    1. Enable/disable hyperthreading in your BIOS.
    2. Run benchmarks on the same workload with hyperthreading on and off.
    3. Compare CPU utilization and overall performance metrics.

Challenge 8: Analyze CPU Resource Limits​

  • Objective: Understand the impact of setting resource limits on CPU usage.
  • Instructions:
    1. Use ulimit to set CPU time limits for processes.
    2. Run a CPU-intensive job and monitor its behavior as it hits limits.
    3. Analyze how the system handles job termination and resource cleanup.

Challenge 9: Investigate Real-time Task Scheduling​

  • Objective: Explore how real-time tasks interact with regular tasks.
  • Instructions:
    1. Create a real-time task using pthread and set its scheduling policy.
    2. Run it alongside regular tasks and measure impact on responsiveness.
    3. Analyze any priority inversion or scheduling anomalies.

Challenge 10: Profile Application Performance​

  • Objective: Use profiling to identify bottlenecks in CPU-bound applications.
  • Instructions:
    1. Use gprof or valgrind with callgrind to profile an application.
    2. Analyze the generated reports to find hotspots and optimize the code.
    3. Measure performance improvements after optimizations.